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Executive summary 

In 1996, the Brazilian Ministry of Education introduced a National Assessment of Courses for 
Brazilian higher education. The assessment consisted of a test applied to all students graduating from 
specific course programs in the country. The results were published on a five-point scale, from A to E, 
according to their distribution in each field. In the first year, the test was applied to students who were 
graduating from the programs with the largest attendance: Law, Administration and Civil Engineering. 
In 2003, the exam was expected to include 470 thousand students graduating in 26 different fields in 
6,500 course programs in the whole country. 

The objective of the test was to provide information to the public on the quality of higher 
education courses, helping the students and their families to choose where to study, and to provide the 
Ministry of Education with information that could be used in the accreditation and reaccredidation of 
higher education institutions. Besides, the exam generated an intensive process of discussion and 
consultations among academics about the contents and standards of the different careers, which 
supposedly helped to improve the quality of Brazilian higher education throughout. 

The exam was introduced without previous consultation, and was received with strong 
opposition from student associations, teachers' unions and many higher education institutions. 
However, from the beginning, it received strong support in public opinion and in the press. The 
criticism ranged from specific objections to the way the tests were conceived and the results presented 
- a uniform test for the whole country, a national rank of outcomes without consideration of existing 
conditions and explicit standards - to broad objections to any kind of measurement of education 
outcomes. However, once in place, the results became widely used by students in their choice of 
institutions, and by institutions themselves, particularly in the private sector, to publicize their results, 
or to try to improve them. Bad results, when persistent and associated with other indications of low 
quality, were supposed to lead to the closing down of the course programs by the education authorities, 
but, in practice, this has seldom happened. 

In 2002 the opposition Laborer's Party won the Presidential Elections and the candidate's 
program for education announced the end of the National Exam. Once in power, the new Minister of 
Education established a Commission to examine the issue and to propose a new approach to higher 
education assessment. The Commission published its conclusions in September 2003 and in December 
the Government announced his own proposal for higher education assessment, which changes the 
previous system substantially. 
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Introduction 1 

Brazilian higher education developed late, and was based on the European, mostly French and 
Italian models. Until the early 19 th century, Brazil was a Portuguese colony and no higher education 
institutions existed. In order to get a degree, one had to go to Coimbra in Portugal or perhaps to France. 
In 1808 the Portuguese King and his court moved to Brazil, fleeing from the invading Napoleonic 
troops, and Rio de Janeiro became, for several years, the capital of the Portuguese Empire, to become 
later an independent country. The first higher education institutions were established in those years - 
one military academy, later to become a school of engineering; two medical schools; and two law 
schools. They were all owned, financed, controlled and supervised by the national government. In the 
late 19 th and early 20 th century, as the old Brazilian Empire was replaced by a decentralized Republic, 
other institutions were added. Some states - notably the state of Sao Paulo - started to create their own 
institutions, and private institutions began to appear. Until 1889, only 24 higher education schools 
existed; between 1889 and 1918, 56 new, mostly private, higher education schools were established. 2 
New fields, like pharmacy, dentistry, agriculture, and accounting, were introduced side by side with 
the old learned professions. 

The first universities were established in the 1930s, and they were, mostly, a collection of old 
schools, or faculties, with one important innovation, a new Faculty of Philosophy, Sciences and 
Letters, which was to be, at the same time, the place for scientific and academic research, and for the 
preparation of secondary school teachers. The first university, the University of Sao Paulo, was 
established by the State government inl934, and the Universidade do Brasil, now the Federal 
University of Rio de Janeiro, was established in 1939 by the National government. In the early forties, 
the Catholic Church created the first private university in Rio de Janeiro, and they all introduced 
course programs in the natural sciences, mathematics, history, geography, social sciences, philosophy, 
language and literature, which did not exist before. 

Throughout the 19 th century, holders of higher education degrees strived to assert their 
exclusive rights to practice their respective professions, and, after the 1930s, the principle that a 
university degree was tantamount to a professional license became firmly entrenched. 3 This created, at 
once, a problem of regulation, which was never fully solved. To deal with this, a new Ministry of 



1 1 am indebted to Gilda Portugal, Maria Helena Guimaraes Castro, and, particularly, to Jocimar Archangelo, for the 
wealth of insight and information on the National Assessment of Courses. I am also grateful to Claudio de Moura Castro, 
Jacques Schwartzman and Maria Helena Magalhaes Castro for detailed comments and criticisms on the first draft of this 
paper. 

2 For the early history of Brazilian higher education, see Fernando de Azevedo, Brazilian culture; an introduction to the 
study of culture in Brazil (New York,, 1971), Eunice Ribeiro Durham, "Higher education in Brazil - public and private," in 
The Challenges of Education in Brazil, ed. Colin Brock and Simon Schwartzman (Oxford, UK, forthcoming), Simon 
Schwartzman, A space for science the development of the scientific community in Brazil (University Park, 1991), Anisio 
Teixeira, Ensino superior no Brasil: analise e interpretagao de sua evoluqao ate 1969 (Rio de Janeiro, 1969). 

3 For this evolution, see Edmundo Campos Coelho, As Profissoes imperials: advocacia, medicina e engenharia no Rio 
de Janeiro, 1822-1930 (Rio de Janeiro, 1999). 
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Education was established, together with a National Education Council, formed by public 
personalities. The new ministry tried to establish a "model university" in the country's capital, based 
on a detailed description of the course contents of all disciplines, down to the assignation of textbooks 
and time tables, which all other institutions had to follow 4 . At the same time, the government created a 
complex system of professional councils, which, together with the business associations and the trade 
unions, were supposed to organize the country into a neat and coherent corporatist structure, 
integrating the professions, the entrepreneurs, the unions and the education institutions. 5 

This tightly conceived system never worked in practice, and its limitations became all too 
obvious as higher education began to expand and new professions started to emerge after the Second 
World War. However, the basic assumptions established in the 1930s - that all higher education 
degrees should be equivalent to a professional certification, that all professions had to be regulated by 
law, controlled and supervised by a legally established professional council or association, and that it 
was the role of the Federal government to make sure that all course programs provided equivalent 
contents - remained and are still in place. The National Education Law of 1996 introduced more 
flexibility, and the legal requirement that each career should have a national "minimum curriculum" 
was replaced by more general "curriculum guidelines". 6 In a sense, the National Assessment of 
Courses of the 1990s could be seen as a step backwards in terms of centralization, although, in many 
cases, there was a genuine effort to limit the assessment to very central skills and competencies, 
allowing for local experimentation and variations. 

The policy problem 

In the forties and fifties, the Federal government created a network of Federal Universities. 
These Federal universities were established usually by the absorption, through legislation, of existing 
private and state-based institutions, based on political considerations, without any mechanisms of 
quality assurance. At the same time, new private institutions emerged, first as religious and community 
based institutions, and later, predominantly, as profit-oriented endeavors. In 1968, there was an 
important university reform, introducing several innovations taken from the American context - post 
graduate degrees, the credit system, departments and institutes - with the assumption that all higher 
education should evolve towards a university model, based on academic research and a full-time 
academic profession. Simultaneously, however, the government responded to the growing demand for 
higher education by making it easier for private institutions to open up and offer degrees, without too 
much control and oversight. By the 1990s, higher education in Brazil had expanded very rapidly. The 



4 Simon Schwartzman, Helena Maria Bousquet Bomeny, and Vanda Maria Ribeiro Costa, Tempos de Capanema, 2 ed. 
(Sao Paulo, Rio de Janeiro, 2000). 

5 James M Malloy, ed., Authoritarianism and corporatism in Latin America, Pitt Latin American series (Pittsburgh, 
1977), Phillipe C Schmitter, "Still the century of corporatism?," in The new corporatism social-political structures in the 
Iberian world, ed. Fredrick B Pike and Thomas Stritch (South Bend, Ind, 1974), Simon Schwartzman, Bases do 
autoritarismo brasileiro, 3a ed., Contribuicoes em ciencias sociais; 10 (Rio de Janeiro, 1988). 

6 Nina Beatriz Ranieri, Educagao superior, direito e Estado na Lei de Diretrizes e Bases (Lei no. 9.394/96) (Sao Paulo, 
2000). 
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number of students doubled in ten years, from 1.5 to more than 3 million, two thirds of them in private 
institutions. Some of these institutions tried to follow the 1968 model of university organization. Most 
of them, however, provided just one or a few undergraduate programs, mostly in business 
administration or law, and relied on part-time lecturers, drawn from the professions or from retired or 
moonlighting academics from the public sector. 

The pressure for and against opening up new institutions and controlling their quality comes 
from many sides 7 . Brazil's higher education coverage, at about 15% of the 18-24 age cohort, is still 
very limited, and the social and economic benefits of higher education degrees and the entrance in the 
learned professions are very high, creating a growing demand for more places. In recent years, the 
provision of private higher education became a multi-billion dollar business, employing about 200 
thousand people, including both academic and administrative personnel. 8 Side by side with small 
institutions, there are now very large private universities, with tens of thousands of students in many 
different locations, with considerable ability to lobby the government and congress for freedom from 
control and regulation. Opposition to the expansion comes from the professional organizations, 
particularly in Medicine and Law, who are concerned about the watering down of their professional 
standards and job market privileges. These concerns are shared by academics and students in public 
institutions. 

Quality assurance is not, however, a problem limited to the private sector. The Brazilian 
legislation grants full academic autonomy to universities, many of them public, which includes the 
right to create new course programs and to define the number of students admitted each year. The 
assumption is that universities are established according to strict academic standards, but in fact public 
universities can be created by Federal or state legislative acts. In principle, private institutions need to 
be accredited to get university status and be granted the same autonomy, but, in practice, accreditation 
has been granted case by case, without any systematic assessment. A new type of institution has been 
officially recognized in recent years. "University centers" are mostly private institutions, dedicated 
solely to teaching, presumably of good quality, and have almost the same autonomy as universities 9 . 
Thus, the authority of the Ministry of Education is limited to approving new universities and university 
centers in the private sector, and to the minute oversight of non-university institutions, which have to 
apply for each new course they want to establish, and for the number of students they expect to admit. 

The demands for a system of quality assurance, beyond the bureaucratic and ineffective 
procedures of the Ministry and the National Council of Education, has been clear since at least the 
Presidential paper on Higher Education of 1985. 10 The paper has led to several initiatives including a 



7 Simon Schwartzman, Higher education in Brazil the stakeholders, LCSHD Paper Series (Washington, DC, 1998). 

8 Jacques Schwartzman and Simon Schwartzman, "O ensino superior privado como setor economico," Ensaio - 
Avaliacao e Politicas Piiblicas em Educacao 10 (out-dez, 2002). 

9 By the end of 2003, the Brazilian government issued a Decree that forbids the creation of new university centers, and 
sets a time limit for their transformation into universities or reversion to non-autonomous status. Brasil Presidencia da 
Republica, "Dispoe sobre os centros universitarios de que trata o art. 11 do Decreto n.° 3.860, de 9 de julho de 2001, e da 
outras providencias," Didrio Oficial da Uniao, Dec 12 2003. 

10 Ministerio da Educacao and Comissao Nacional para a Reformulacao da Educacao Superior, Uma nova politico para a 
educagao superior brasileira - Relatorio Final, Simon Schwartzman (relator) ed. (Fortaleza, 1985). 
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program to provide universities with resources for their self-evaluation 11 and the establishment of 
National Commissions of Specialists to define and revise the minimum core curricula of the different 
careers. In late 1995, under Minister of Education Paulo Renato de Souza, a comprehensive system of 
assessment of higher education was created. It included the development of a yearly census, to provide 
quantitative information on the sector by region, state, fields of knowledge and type of institution; 
qualitative assessments of each institution, large and small, looking at their installations, institutional 
development plans, research performance and other indicators of quality; and assessment of individual 
course programs or careers. The assessment consisted of two components. The first component was an 
assessment of resources, in terms of academic personnel, infrastructure, and internal organization (if 
they had clearly defined mission, self assessment, and coherent pedagogical projects. This assessment 
was carried out by peers, who visited each course program to get the information and processed the 
information according to a pre-defined template. The second component was the National Assessment 
of Courses — an exam all students had to take before graduation. Post-graduate education (Masters and 
Doctoral programs) have been subject to a well established assessment procedure which remained in 
place. 12 



Implementation 

The authority for the Ministry of Education to implement the assessment was established by 
federal law, 13 which makes it mandatory for students to complete the test if it is applied to their field 
in their last year of studies, as a precondition for their degrees. This was possible because higher 
education degrees in Brazil, to be legally valid, have to be registered with the Ministry of Education, 
usually through the office of a Federal university. However, there is no minimum pass grade for the 
students, since the goal is to assess the course program, not the student. In the first years, the National 
Student Union asked their members to boycott the exam, and, in some institutions, the students would 
just sit without answering the questions. This however led to a low ranking of their course, which 
reflected badly on their colleagues who did participate, and this practice was abandoned almost 
completely in the following years. 

The National Assessment is implemented by an agency within the Ministry of Education, the 
National Institute for Education Research (INEP), according to a very elaborate procedure. 14 First, an 
assessment committee is established for each field of knowledge. Members are chosen from lists 
prepared by professional associations, teaching and scientific associations, and by the Brazilian 



11 Ministerio da Educacao and Secretaria de Educacao Superior, Programa de Avaliacao Institucional - PAIUB (1997 
[cited); available from http://bve.cibec.inep. gov.br/ac_rap. asp?cat=21&nome=Avaliacao%20da%20Educacao%20Superior. 
For a discussion of this program, see Alberto Amaral and Marlis Polidori, "Quality evaluation in Brazil: a competency 
based approach?," Higher Education Policy 12 (1999). 

12 Ministerio da Educacao, 'Avaliacao e informacao como instrumento de polltica educacional," in Educagao - Politicas 
e Resultados (Brasilia, 2002). 

13 Federal law 9131/95. 

14 See, for a detailed description, INEP, Provao 2002 - Relatorio Sintese - Resumo Tecnico (Brasilia, 2002). 
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Council of Rectors and the Ministry of Education. They should also be representative of Brazil's 
different regions, and different types of institutions - public and private, large and small. From these 
lists, the Ministry of Education chooses seven names in each area. Thus, for the year 2002, there were 
24 such commissions: Administration, Law, Civil Engineering, Chemical Engineering, Veterinary, 
Dentistry, Electric Engineering, Journalism, Language and Literature, Mathematics, Economics, 
Mechanical Engineering, Medicine, Agronomy, Biology, Physics, Psychology, Chemistry, Pharmacy, 
Pedagogy, Architecture, Accounting, Nursing, and History. They meet in Brasilia, and their task is to 
define the general contents, scope and goals of the assessment of their fields. For their work, the 
Ministry obtains all course descriptions, pedagogical projects and teaching programs adopted by all 
institutions in the country, and organizes this material in terms of their goals, objectives, basic 
bibliography, teaching procedures, and so on, identifying eventual differences in these orientations and 
goals. The Commissions also use assessment reports from previous years, which are prepared by the 
Ministry with course coordinators and professors in each field. Based on this information, the 
Commissions each year revise and improve the guidelines of the previous year, in an interactive and 
continuous learning process. 

Once ready, the guidelines prepared by the Committee are passed on to an external contractor, 
who has the responsibility of developing the tests, administering them, and tabulating the results. The 
choice of this external contract is made through open, competitive bids. In practice, two institutions 
working together, the Fundacao Carlos Chagas in Sao Paulo and Fundacao Cesgranrio in Rio de 
Janeiro, have won all these bids since 1995. They are experienced in administering large-scale 
assessments, having started with the entrance examinations for public institutions in Sao Paulo and Rio 
de Janeiro. They also recruit academic in the universities to develop the instruments, and persons in 
different institutions to deliver, control and oversee the exams. 

Before the exam, the institutions have to provide a list of all students likely to complete their 
course programs in the given year. The exam takes place on the same day throughout the country and 
is widely announced in the press. Observers from professional associations, teaching associations and 
other entities are regularly invited to be present in the different locations where the exam takes place. 

Just before the exam, the students receive from the Ministry of Education a magazine, 
explaining the purposes of the exam, the description of the procedures, and other materials. The core 
instrument is a written exam, which can be either a multiple choice test, or open ended questions, or 
both, according to the Committee's recommendations. The general orientation is to put emphasis on 
the mastery of key concepts, on the ability to think independently and to apply knowledge to new 
situations; rote learning and the accumulation of information for its own sake are discouraged. Another 
instrument is a survey questionnaire, in which the students are asked to provide socioeconomic 
information on themselves and their families, and their views and perceptions about their course 
programs. A third instrument is the student's assessment of the assessment - if they like the 
instrument, if they considered it too easy or too simple, inappropriate, etc. 

Multiple-choice tests are marked using optical scan technology and grades are provided after 
the Commission assesses each item's discrimination, level of difficulty and reliability. For open-ended 
questions, a sample of the responses is used to develop an assessment protocol, which is then applied 
to the universe of respondents. The grade that a student receives is based on his or her relative place in 
the distribution of results for the whole country. According to the mean results of their students, each 
course program receives a grade from A to E. The students are confidentially informed about their 
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individual result in a bulletin with information about their relative placement in their class, region and 
the country. The course's mean score, however, are made public. 

There are several follow-ups, besides the establishment of the grades. Immediately after the 
exam, the correct answers to the questions are made public, so that the students can see what they did 
right or wrong, and the professionals in the field can assess the quality of the exam. Then the aggregate 
results of the students' evaluation of the assessment for each course are made available to the course 
coordinators on the Internet. 

The next step is a series of national seminars, for each field of knowledge, to discuss the results 
of the last exam, with the cooperation of professional associations, course coordinators and 
universities. In these seminars, the Commissions present the results and their views, complaints are 
aired, and the officers from INEP in charge of the whole process have an opportunity to hear the views 
of the academic community and express their perceptions of the whole process. 

Meanwhile, the Ministry prepares a series of technical reports about the exam, and also 
summarizes data from the socioeconomic questionnaire, which provides information on characteristics 
and attitudes of the students. The technical reports include a synthesis of all the results, reports for each 
field of knowledge, and individual reports sent to the persons in charge of each course program. 

Finally, some research institutions and independent researchers are asked to conduct more in 
depth analysis of the data, which may be used and disseminated by the Ministry, published as 
academic papers, or remain as technical reports of limited circulation. 15 



Costs 



There is no estimation of the total cost of the operation. In 2002, the cost paid to the external 
contractor was about 36 million reais, or 12 million US dollars. With these resources, the contractor 
was required to prepare 24 different exams to be applied to 361, 000 students graduating from five 
thousand course programs in 627 municipalities. The per capita cost was, therefore, one hundred reais, 
or 33 dollars per student. There are many more course programs in the country, but these 24 account 
for about 90% of the students graduating in that year. There is no information about the internal costs 
for the Ministry of Education, which includes travel of the 168 members of the academic commissions 
to meetings in Brasilia, the time of the staff working in the preparation of the materials for the 
Commissions to work, the organization of seminars and other events, and contracts with external 
consultants for the analysis of the data. It is a sizeable effort, but not out of proportion considering that 
the Ministry of Education spends about five billion reais - 1.6 billion US dollars - a year on higher 
education alone. 



15 The Center for Studies of Public Opinion of the University of Campinas was commissioned to prepare of these reports, 
which do not seem to be confidential, but was not widely distributed. See Rachel Meneguello et al., "Relatorio Final, Meta 
4 - Analise dos dados do Exame Nacional de Cursos -Provao," in Projeto Estudos Socioeconomicos em Educacao 
(Campinas, SP, 2002). 
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Impact 

The Brazilian legislation gives to the Minister of Education, with the support of the National 
Council of Education, the authority to accredit new higher education institutions, and to renew their 
accreditation periodically. In practice, however, once a higher education institution has been allowed to 
function, only in extreme cases will it lose its authorization or accreditation, and the process of 
periodical accreditation and reaccredidation of universities was never fully implemented. The 
government has intervened in a few private institutions in recent years, but never in a public university, 
and never because of a negative assessment of their academic quality (in some cases, attempts by the 
Ministry to close down bad quality courses and institutions were stopped by the judiciary, or by 
appeals to the National Council of Education). The National Assessment of Courses was meant to 
provide important information for such decisions, but since the results refer to course programs, and 
not to whole institutions, they can be at most one element of information in a much broader assessment 
procedure, which is to be still implemented. 

Because of this, the direct contribution of the National Assessment for the regulation of higher 
education has been minimal. Its indirect impact, however, is considered very important. One such 
impact was to encourage the students to search for better ranked course programs. According to a 
study done by the Ministry of Education, the number of new applicants for courses in Administration, 
Law, Civil Engineering, Chemical Engineering and Dentistry, who received "D" and "E" in the 
assessment, went from 35 to 18 thousand between 1997 and 2001, while the demand for courses rated 
"A" increased by 6%. Another finding was that new courses in private institutions, established after 
the assessment was introduced, tend to be better than many old ones. Thus, both students and academic 
officers are taking the concepts into account, and changing their behavior. Students look for better 
course programs and universities work to higher standards. 16 

The professional and academic associations in Administration carried out a detailed survey 
among course coordinators on the impact of the assessment in their institutions. 17 They asked about 
changes introduced in the course programs in the last three years, and whether these changes were 
induced by the assessment or not. About 65% of the course programs reported changes in the period, 
half of which were attributed directly to the national assessment. In general, private institutions reacted 
more to the assessment than public ones, but the difference is not large - 38 vs 30% of all the course 
programs in the sample. Not surprisingly, the most frequent innovation was to prepare students for the 
exam, followed by changes in pedagogical and teaching practices of different kinds. Changes 
involving investments, infrastructure, and salary raises were much less frequent. 



16 Ministerio da Educacao and Instituto Nacional de Estudos e Pesquisas Educacionais, Cai demanda por cursos com 
baixo desempenho no Provao (2002 [cited November 5, 2003]); available from 
http://www.inep.gov.br/imprensa/noticias/edusuperior/provao/news02_27.htm. 

17 Conselho Federal de Administracao, Associacao Nacional de Cursos de Graduacao em Administracao, and Ad 
Homines, Alteracoes, efeitos e influencias do provao de administracao na opiniao dos coordenadores dos cursos de 
administragao (Brasilia, 2003). 
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Main changes in administration courses induced by the National Assessment of Courses 

% of coordinators 
reporting the change 

Using question items of the assessment in classroom 82.2 

Changing teaching methods 68.3 

Changing course contents 66.8 

Assessing the students abilities to perform in the exam 61.4 

Intedisciplinary work 55.5 

Mock assessments 55.5 

Improving the library 54.5 

Upgrading the teaching staff 52.0 

Working to improve the image of the institution 51.5 

Improving the use of the library by the students 47.0 

Strategic planning 41.1 

Marketing 41.1 

Hiring new staff 39.1 

New multimedia resources 35.6 

Internet access 34.7 

Teacher training 33.7 

Links with firms and business sectors 32.2 

Trainee progams for students 32.2 

Investments in computers 31.7 

Interactions with the community 29.7 

More working time for staff 25.7 

Better equipment in classrooms 25.3 

More fellowships for students 13.4 

Higher salaries for academic staff 8.0 

The few studies that exist on the socioeconomic characteristics of the students, with the 
information produced by the exams socioeconomic questionnaire, provide very interesting information, 
some of it unexpected. 18 In general, achievement has much more to do with the characteristics of the 
institutions than with the characteristics of the students, and the correlation between socioeconomic 
status and achievement is not high. Part of the reason is that course programs in the private sector tend 
to be of lower quality than those in the public sector, but students in the private sector come from 
families with higher income than those in public institutions. The other reason is that once the students 
are able to reach higher education, they have already overcome most of the disadvantages that would 
usually affect their academic performance. There are however important differences in careers choice. 
The parents of more than half of the students in journalism, law, engineering and medicine have a 
higher education degree, against less than 20% of those in teaching careers (mathematics, language) 
and less than 10% of those in pedagogy. There are some differences among public and private 
institutions, but they are much less significant than those among careers. Finally, detailed regression 
analysis confirms that achievement depends above all on whether the student is enrolled in a public or 
private institution, and on factors like age, knowledge of English, hours dedicated to study, work, and 



18 Meneguello et al., "Relatorio Final, Meta 4 - Analise dos dados do Exame Nacional de Cursos -Provao.", Helena 
Sampaio, Fernando Limongi, and Haroldo Torres, Eqiiidade e heterogeneidade no ensino superior brasileiro (Brasilia, 
2000). 
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10 



whether the student attended public or private secondary education (with best results for those coming 
from private schools). 

One of the most important contributions of the National Assessment, not readily documented 
but very clear in the minds of those responsible for its implementation, was the opportunity it provided 
for course coordinators, academics and professional associations to come together in a continuous 
process of discussion and negotiation about the quality standards of their respective fields. Beyond the 
efforts of many institutions to "learn the tricks" of the exam to get better grades, there are many stories 
of institutions looking for help to improve their courses, and others closing down because of the lack of 
student demand. 



Opposition and criticism 

From the onset, the National Assessment was received with strong opposition from the 
National Students Union (UNE) and some public universities. The Student Union asked students to 
boycott the exam and tried to disrupt its implementation. Both the Student Union and the Federal 
University of Rio de Janeiro went to Court trying to stop the assessment from taking place. The 
students argued, among other things, that the assessment would hurt the students from the institutions 
receiving lower ratings. The eventual shortcomings however were not the responsibility of students, 
but of the institutions, or the government, which did not provide the institutions with the support they 
needed. The arguments coming from public universities were similar. If they did not perform well, it 
was because they were not getting the necessary support, and should not be punished for that. There 
were other criticisms, from general statements about the impossibility of measuring and quantifying 
quality, to a principled stand against bringing a market mentality to the realm of culture and education 
by establishing comparisons and competition among institutions and students,. 

These criticisms have to be placed in the Brazilian political context of the time. Both the 
National Student Union and the higher education teachers' association were in the opposition to the 
Fernando Henrique Cardoso government, and strong critics of whatever initiative came from the 
Ministry of Education, for good or bad reasons. Cardoso, a renowned sociologist and former professor 
at the University of Sao Paulo, had a history of strong opposition to the Brazilian military regime that 
lasted until 1985. He was elected President in 1994 after being able, as Brazil's economic minister, to 
bring the country's inflation under control. His mandate, which lasted until 2002, was characterized by 
very significant efforts to bring order to the economy and reduce the runaway expenses of the public 
sector. It was a period of economic stagnation and his government was accused of obeying the 
neoliberal orientations of the International Monetary Fund. One of the strongholds of the opposition 
was the organized civil servant unions, which included employees of public universities. 19 

Partisan reasons aside, several criticisms to the assessment are reasonable. The adoption of a 
single, unified exam for all course programs in the country led all the institutions to adjust to the same 
mold, and may have thwarted their freedom to experiment and to diversify. By selecting a group of 



19 See, on the period, Fernando Henrique Cardoso and Mauricio A Font, Charting a new course the politics of 
globalization and social transformation (Lanham, MD, 2001), Mauricio A. Font, Transforming Brazil : a reform era in 
perspective (Lanham, Md., 2003). 
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specialists to write up the exam, the Ministry made the particular bias of this group the national 
standard. This policy was consistent with the traditional view that all higher education course programs 
in a given field should provide the same contents and equivalent professional certifications. In areas 
with well established academic and professional paradigms this is not controversial. But this is the 
exception, rather than the rule, in a highly differentiated mass higher education system with different 
types of students, institutions, and visions about what the contents of higher education should be. 

The decision to publicize the rank of each course on the five-point scale, based on the 
distribution of results, was a conscious choice not to establish clear reference, or cutting points, in 
relation to which a given course program could be considered acceptable or not acceptable. So, in a 
field where all course programs are of very good quality, 12% of them would be ranked as "E", while 
in another, where all course programs are bad, 12% would receive an "A" 20 . In other words, all courses 
are ranked by uniform criteria, and the public is informed about their relative position in the rank, but 
not if they are of good quality or substandard. The reason for this was never spelled out very clearly, 
but it is not difficult to understand. The establishment of cutting points would be very controversial, 
and the official information that many, perhaps the majority, of the course programs in many fields are 
substandard - a very likely result - would lead to a crisis the Ministry could not possible handle. 

A third criticism is that the assessment may be measuring the cultural capital the students bring 
to the university, rather than the education value added to them by their courses. Prestigious 
institutions attracting very good students would have good results even if the courses were bad; 
hardworking and dedicated institutions accepting students with poor backgrounds would not be able to 
get higher marks, regardless of their effort. It would be possible to estimate the value added by the 
courses by taking into account the student's achievements on their entrance examinations to the 
university, or their achievements in a another national voluntary test, applied o students at the end of 
secondary school. A statistical analysis using information from the student's university entrance 
examinations in the state of Minas Gerais shows that, indeed, previous conditions affect the final 
outcome, but that, in general, this information would not change the final rankings in the national 
assessment, except in a few isolated cases. 21 

A fourth criticism was that by looking only at the students' results, without considering 
variables related to the academic staff, equipment, computer facilities, library resources, and so on, the 
National Assessment was at most only a partial assessment instrument. In fact, in addition to the exam, 
the Ministry of Education developed another assessment procedure for these input variables. A heavy 
weight was given to the academic degrees of the faculty (the percentage holding doctoral and master 
degrees) and to the percentage with full time contracts, plus to an assessment of the physical 
equipment and pedagogical projects. Initially the Ministry ranked the course programs according to a 
combination of these instruments. The information on inputs is necessary and useful, but there are 
good reasons not to combine input and output effects on the same scale. It is important to know, for 



20 Until 2000, the grades were distributed according to fixed percentages - 12%, 18%, 40%, 18% and 12%, for A, B, C, 
D, E. Since 2001, the normal distribution was used, with courses above one standard from the mean receiving an A, and 
those one standard deviation below receiving an E. 

21 Jose Francisco Soares, Leandro Molhano Ribeiro, and Claudio de Moura Castro, "Valor agregado de instituicoes de 
ensino superior em Minas Gerais para os cursos de direito, administracao e engenharia civil," Dados - Revista de Ciencias 
Sociais 44 (2001, 2001). 
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instance, which inputs are more effective than others in producing the outcomes. Besides, most 
lecturers in public institutions are nominally full-time, while most in the private sector are not, and this 
introduced a bias in favor of public institutions. 

The Commission established by the Ministry of Education in 2003 to propose a new national 
assessment system for higher education in Brazil presented a detailed criticism of the current Course 
Assessment and suggested a different path. Some of the criticism was technical: the lack of clear 
standards and the measurement of the educational value added from the courses, and the lack of 
comparability of results over time. Other criticism was more general: the exams responded to 
"motivations coming from outside, rather than inside the institutions, leading to isolated distorted and 
wrong representation of the academic world", or that "its rationality was much more market oriented 
("mercadologica") and regulatory than academic and pedagogic". Other criticisms were related to the 
growing cost of the assessments. According to the report, the current costs are likely to grow, as higher 
education expands and new fields and disciplines are included in the assessment. 22 



The new higher education assessment 

With the change of government in early 2003, the original team responsible for the 
establishment and implementation of the National Assessment within the Ministry of Education was 
disbanded, and most of the institutional memory and experiences accumulated in recent years has 
already been lost. In 2003, the Ministry of Education went ahead with the implementation of the exam, 
following the standing legislation, but without carrying on the usual procedures of analyzing the results 
with the participation of the academic committees. In December of 2003, at last, the government issued 
a "provisional act 23 " changing the legislation regulating the whole higher education assessment system, 
while the Ministry of Education issued another document spelling out how it intended to proceed. The 
provisional act created a new system for the assessment of higher education, based on two new 
National Commissions, one to provide guidelines and another to implement the new procedures. The 
members of both commissions are to be nominated by the government. The first commission will 
consist of persons with recognized competencies and representatives of the "organized civil society" of 
students, teaching and administrative staff, and the second will consist of civil servants from the 
Ministry of Education. 

The new system is supposed to rank the "institutional quality" of higher education 
establishments on three levels: satisfactory, regular, and not satisfactory. The existing five-level 
ranking system disappears and the new ranking will combine the results of four different assessments: 



22 Comissao Especial da Avaliacao da Educacao Superior, Sistema Nacional de Avaliacao da Educacao Superior 
SINAES: Bases para uma nova proposta de avaliacao da educacao superior (2003 [cited September 5, 2003); available from 
http://www.funadesp.org.br/downloads/Nova_Proposta.pdf. p,. 60. 

23 Brasil Presidencia da Repiiblica, "Medida Provisoria 147, de 15 de dezembro de 2003. Institui o Sistema Nacional de 
Avaliacao e Progresso do Ensino Superior e dispoe sobre a avaliacao do ensino superior," Didrio Oficial da Unido, 
December 16 2003. In Brazil, it is possible for the Executive branch to create laws through provisional acts ("Medidas 
Provisorias") which are valid immediately, but can be changed or rejected by Congress within a short limit of time. This is 
supposed to be used only on extraordinary situations, but in practice it is used whenever the government wants to avoid the 
lengthy procedures of sending ordinary bills to be discussed by Congress. 
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institutional capabilities, teaching, knowledge production processes (presumably research), and social 
responsibility. In another document the Ministry of Education spelled out the broad outlines of the new 
assessment it expects to undertake. 24 There will be an "Index of Development of Higher Education" 
(inspired by the Index of Human Development of the United Nations Development Program), which 
will combine the results of the four assessments. The National Assessment of Courses remains, to 
assess learning process. But, instead of a yearly universal assessment of all graduating students and 
course programs in specific fields, the assessments will be done now every three years, through 
sampling procedures. And, instead of just one assessment, there will be two, one at the beginning, the 
other at the end of the course program. 

The former Minister of Education, Paulo Renato de Souza 25 , in a press conference, indicated 
some potential problems, stating that, in practice, the government is shutting the assessment system 
down. Other observers are also raising questions. By making the participation in the assessment 
voluntary for the students, the Ministry may not be able to get them to participate; the proposed 
sampling procedures have not been spelled out; it is not clear how the assessment of a sample of course 
programs could be combined with the assessment of institutions; it is not clear whether the assessments 
to be published will refer to course programs or to institutions as a whole, which seems to be the case; 
the new legislation bypasses the National Council of Education; and the new evaluation committees 
are likely to represent the existing unions of students, lecturers and civil servants, rather than the 
academic and professional communities of the country. Finally, by combining the results of the 
assessment of outputs with three other assessments, supposedly with the same weight, the new 
procedure is likely to obscure, for society, the main information it wants, the quality of the education 
provided in specific course programs, which may vary widely within the same institution. In spite of 
this criticism, the new system is being presented as an important improvement over the past, and it is 
necessary to wait and see how it will be implemented. 



24 Ministerio da Educacao, Sistema Nacional de Avaliagao e Progresso da Educagao Superior: Uma nova sistemdtica de 
avaliagao do Ensino Superior brasileiro (2003). 

25 Paulo Renato Souza, "Observagoes sobre a Medida Provisoria 147 Sobre o sistema de avaliagao do ensino superior," 
(2003). 
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The future 

The Brazilian National Assessment of Courses is a unique and extraordinary experience, which 
has generated admiration and interest in higher education circles in different parts of the world, and 
received strong support in the Brazilian public opinion. Its future, however, is uncertain. On hindsight, 
it is possible to say that the main weakness of the National Course Exam was its lack of proper 
institutionalization and the absence of a clear sense of ownership within Brazil's higher education and 
professional communities. The Exam started as a personal initiative of the Minister of Education, 
Paulo Renato de Souza (an economist who had been the rector of the University of Campinas and a 
high ranking officer of the Inter American Development Bank) who had to start by convincing his own 
staff of its need. Its implementation was assigned to the most flexible and independent branch of the 
Ministry of Education, the National Institute for Education Research (INEP). In principle, other 
institutions could have taken this task — the Secretary for Higher Education within the Ministry, the 
National Council of Education, the National Conference of Rectors — and a new institution could have 
been created for this purpose, like the National Commission for the Assessment and Accreditation of 
Universities in Argentina. 26 

Had the Minister decided to work through one of these institutions, subject to all kinds of 
interest groups and administrative hurdles, or to create a new one, he might not have succeeded in 
moving so rapidly, and achieving so many significant results in such a short time. Acting on the power 
of his cabinet and thanks to his personal prestige, it was possible to move quickly through the complex 
legislative process to get the legal authorization and to place the necessary human and financial 
resources in the hands of the able head of INEP, Maria Helena Guimaraes Castro. 

The price, however, was that no institution or segment of the academic community claimed 
ownership of the Assessment, except a small team within the Ministry of Education. Hundreds of 
academics were asked to participate in the Commissions and probably did a very important work, but 
they were there by the Minister's invitation. The statistical data generated by the exams remained 
under the Ministry's control. Some qualified researchers and research centers were invited to analyze 
the data, but they were not made publicly available to the academic community of education 
researchers. In the effort to keep up with the complex procedures established for the Assessment, most 
of the energy of INEP's staff was dedicated to the preparation of technical documents and other 
materials for the Commissions, the students and the course program coordinators, with little left for the 
deeper reflection on the general importance and significance of the Assessment. To conquer public 
opinion, the Minister had a competent public relation staff, which kept the press well informed about 
the achievements of the Assessment, and helped to win the battle of the public opinion against the 
organized opposition. 

Without clear ownership in society, and being established as just one initiative within a sector 
of the Ministry of Education, the National Assessment did not have the strength to survive the change 
in administration. The new Evaluation Commissions that have been established by the new legislation 
to implement the new assessment system could be a step in the right direction if these Commissions 
can become truly independent and autonomous from vested interests. There is little hope for that, 



' Comision Nacional de Evaluacion y Acreditacion Universitaria (CONEAU). 
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however, given their proposed membership: representatives of the "organized society" - presumably, 
unions of students, lecturers and university employees, known to have been the strongest opponents of 
the National Assessment of Courses as it existed, and functionaries of the Ministry of Education in 
different capacities. 

It is not a promising perspective. To become a stable and significant feature of Brazilian higher 
education, the Assessment would have to find a permanent institutional house, which can be neither 
the Ministry of Education, nor the unions and corporations with vested interests against any kind of 
external assessment of their own work. Between these two extremes, a proper space will have to be 
found, if the experience of recent years is not to be lost. 



Policy Analysis 



PPAQ 



Public Policy for Academic Quality 



16 



Resources for Policy Makers 

a) Live Internet links: 

Persons willing to learn more about the assessment of higher education in Brazil should consult the 
sites of the Ministry of Education in Brazil, and more specially those pages related to Higher 
Education and to the National Institute for Education Statistics (INEP). The site is in Portuguese, and 
its contents reflect the current activities and views of the Brazilian federal education authorities, and 
the site of INEP publishes several kinds of statistical information for all levels of education. The sites 
are: 

• Ministry of Education: http://www.mec.gov.br/ 

• Higher Education: http://www.mec.gov.br/nivemod/educsupe.shtm 

• INEP: http://www.inep.gov.br/ 

b) Technical documents and assessment results 

A selection of technical and other official documents, related to the assessments of 2001, 2002 and 

2003 can be downloaded from the links below. All texts are in Portuguese. 

2003 

Technical report 2003 (Adobe file) 

Results 2003 (Excel file) 
2002 

Technical Report 2002 (Adobe file) 

Results, by fields of knowledge 2002 (Excel file) 

Results, by State 2002 (Excel file) 

Assessment of teaching conditions 2002 (html file) 

Model of Course Program Report 2002 (Adobe file) 

Model of Student report card 2002 (Adobe file) 
2001 

Technical report 2001 (Adobe file) 

Results 2001 by State (Excel file) 

Model of Course Program report 2001 (Adobe file) 

Model of Student report card 2001 (Adobe file) 

New grading procedures (Adobe file) 

c) Studies, press releases, up to 2002 

Castro, Maria Helena Guimaraes, O Exame Nacional de Cursos, Apresentacao ao Seminario sobre 
Educacao e Empregabilidade, Sao Paulo, 4 de maio de 2000 (Adobe file) 

Conselho Federal de Administracao, Associacao Nacional de Cursos de Graduacao em Administracao, 
and Ad Homines. Alteracoes, efeitos e influencias do provao de administracao na opiniao dos 
coordenadores dos cursos de administracao. Brasilia: Conselho Federal de Administracao, 2003. 
(Adobe file) 
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Meneguello, Rachel, Fernando Antonio Lourenco, Jose Roberto Ruz Perez, Plinio Augusto Dentzien, 
Ana Maria Alves Carneiro Silva, Clecio da Silva Ferreira, Fabiola Brigante Del Porto, Fernando Alves 
Silva, Janaina de Rezende Barreto, Joao Henrique Galvao, Jose Vilton Costa, Paula Vanina Cencig, 
Rosilene Sydney Gelape, Sergio Stocco, Simone da Silva Aranha, and Vitor Luiz Cooke Vieira. 
"Relatorio Final, Meta 4 - Analise dos dados do Exame Nacional de Cursos - Provao ." In Projeto 
Estudos Socioeconomicos em Educacao. Campinas, SP: Universidade Estadual de Campinas, Centro 
de Estudos de Opiniao 'Publica, 2002 (Adobe file) 

INEP. ' Alunos apontam melhorias na graduacao ." Press release. Brasilia: Ministerio da Educacao, 
Instituto Nacional de Estudos e Pesquisas Educacionais, 2002 (Adobe file) 

INEP, Cai demanda por cursos com baixo desempenho no Provao 2002. Press release. Brasilia: 
Ministerio da Educacao, Instituto Nacional de Estudos e Pesquisas Educacionais, 2002 (Adobe file) 

Ministerio da Educacao, " O Ensino Superior: Maior e Melhor ." Brasilia: Ministerio da Educacao, 
2002 (Adobe file) 

Soares, Maria Susana Arrosa, Deninse Leite, Maria Auxiliadora Nicolato, and Clarice E Baeta Neves. 
" O sistema de avaliacao do ensino superior no Brasil ." Paper presented at the Encuentro del Instituto 
Internacional para la Educacion Superior en America Latina y el Caribe y Red Iberoamericana para la 
Evaluacion y la Acreditacion de la Calidad de la Educacion Superior, Buenos Aires 2003. (Adobe file) 

Souza, Paulo Renato. " Information y evaluacion como instrumentos de politica educacional ." Estudio 
de Casos preparado para el INDES/BID, basado en la experiencia del Ministerio de Educacion del 
Brasil en el periodo 1995-2002. Version revisada atendiendo a las observaciones del INDES., 2002. 



d) The new assessment system, 2003 

Ministerio de Educacao, Assessoria de Comunicacao Social - Ministro recebe proposta de mudanca na 
avaliacao do ensino superior. SESU, Setembro de 2003 (htm file) 

Ministerio da Educacao, Assessoria de Comunicacao Social. Sistema Nacional de Avaliacao e 
Progresso da Educacao Superior: Uma nova sistematica de avaliacao do Ensino Superior brasileiro, 
2003. (Adobe file) 

Presidencia da Republica, Medida Provisoria 147 , de 15 de dezembro de 2003. Institui o Sistema 
Nacional de Avaliacao e Progresso do Ensino Superior e dispoe sobre a avaliacao do ensino superior. 
(Adobe file) 

Souza, Paulo Renato. "ObservacSes sobre a Medida Provisoria 147 Sobre o sistema de avaliacao do 
ensino superior ." (Adobe file) 
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e) Printed materials available from the Public Policy for Academic Quality Research Program 
(PPAQ), Abernethy Hall, University of North Carolina at Chapel Hill 

Eugenia M. R. Charnet and Maria M. Okuda, Avaliagao das Provas ENC/2002. Relatorio, by 

INEP, Um Estudo sobre o Processo e os Resultados, 2002 

INEP, Manual de Avaliacao Institutional, Centros Universitarios, August, 2002 

INEP, Manual do Avaliador Institutional, 2002 

INEP, Manual de Avaliagao do Curso de Farmacia (2002) 

INEP, Manual para Elaboracao das Provas, 2003 

Relatorio Sintese 1999 

Relatorio Sintese 1999 - Anexo Jornalismo 

Revista do Provao, several volumes, 1966-2002 
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